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ABSTRACT 

Although effective schools research argues that 
careful monitoring, analysis, and discussion of student progress and 
program quality are central to improving education, educators' 
efforts are limited by the dispersion of student data throughout 
district files. The Instructional Decision Support System, used in a 
cooperative venture involving the Wissahickon School District 
(Pennsylvania) and the University of Pennsylvania Graduate School of 
Education, was designed to develop more effective inquiry strategies 
for understanding the relationship between achievement and 
eemographic variables and their effects on student achievement. 
Because the traditional re-uiance on "measures of the average" hides 
more than it reveals, the project attempts to help administrators; av<J 
teachers go beyond averages, rates, and percentages to evaluate both 
student progress and educational program effectiveness. By enabling 
educators to disaggregate individual experience from the total 
population, it is possible to understand how district initiatives 
affect individual students or subgroups. The project developed 
"display analysis" techniques for graphically depicting one or more 
variables for a target population. The displays allow educators to 
analyze the fit between programs, student population, and various 
subgroups. Further analysis can then occur using additional data, 
including staff's personal knowledge about students. (MLH) 
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Close attention to student progress is an important component of 
school improvement activities. Like any bureaucracy, school districts collect 
a great deal of information about student characteristics and performance. 
This data is scattered around the district in a variety of pencil and paper and 
computerized formats. Some of this data finds its way into reports to the 
board, or state and federal educaition agencies. However, the data and the 
reports based on that data are only infrequently available in a form that 
allows teachers and administrators to understand district programs, student 
progress, or student experiences in the district. 

Yet there is an expectation that data readily avc:ilable in a flexible 
format can provide useful insig' t into student educational experiences. The 
question is how to use the data districts normally collect as a flexible 
"database," that is easy to manipulate so that some data could be explored 
while other data are ignored, or explored by comparing variables from different 
sources. 



Over the last three years, we have developed an individual student 
database built on demographic, achievement, and student attitude and classroom 
process variables. This database is a flexible way of storing, organizing, and 
retrieving information on individual students that educators themselves find 
useful in their work and making decisions about how to improve their impact on 
student learning. The prototype of this database is currently being used on a 
limited basis by staff in the district. 

The prototype was developed to meet several criteria drawn from 
management information system, office automation, and information science 
research: 



• the information is provided in a time and form which fits 
users work situations; 

• the information system is well-integrated into their work 
settings so they will use it; 

• the system is flexible so that information can be 
retrieved in a variety of forms and in combination with 
any other information in the system; 

• the system is designed so users can easily add categories 
of data as their needs and interests dictate. 



We began with data the district currently collects, acknowledging 
that data currently being collected might not be the data they ought to be 
collecting. We realized that we would get a better sense of what data we ought 
to collect once we began to work with data already collected and discovered 
what else was needed. 
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PILOT PROJECT 



The purpose of the pilot project was to develop a prototype longi- 
tudinal student database and to test the feasibility of that database as an 
effective and efficient means to disaggregate data connected to achievementr 
demographic variables, and related student experience^ This database consists 
of demographic and achievement variables • Demographic variables included 
student name, student number, sex, race, age, and special program participa- 
tion# Achievement variables included grades and grade point averages, scores 
on district-administered reading tests, scores on standardized achievement 
tests (Metropolitan Achievement Test), and scores on standardized aptitude 
tests (Scholastic Aptitude Test)» 

The results of the pilot project encouraged us to expand our efforts* 
Several Wissahickon School District staff have demonstrated that the database 
can provide them with information which makes them more effective and effi- 
cientt In addition, these ^t^ff have shown that they can develop new instruc- 
tional indicators which increase understanding of the instructional process* 
Other Critical Success Factors include management support and interest in the 
upper and middle ranks, a well-functioning information system, availability of 
longitudinal data, and a district size which provides breadth of experience but 
not so large as to make database development an overwhelming task* 

The pilot project clearly demonstrated the feasibility of developing 
a longitudinal student database that would allow educators to disaggregate from 
summary statistics to look at particular subgroups of students, in addition, 
it demonstrated that disaggregation could provide powerful insights into 
student achievement and educational experience* We discovered, for example, a 
subgroup of students with high grade point averages who didn't score as 
expected on standardized tests* This led us to look for similarities among 
those students, so we might develop interventions appropriate to their needs* 

The graphic representations that were generated using student data 
provided new and unique opportunities for administrators to study student 
experience* Well accepted "rules" about the predictability of achievement were 
shown not to hold true for students in the district* Consequently, eductors 
were compelled to formulate new and better questions about district programs 
and student achievement* The greatest promise offered by the pilot project is 
that we are capable of using existing data, in new combinations, to answer 
questions that previously could not be asked* 



Importance of the Project 

The Instructional Decision Support System is designed to enable 
educators to develop more effective inquiry strategies to understand the 
relationsnip of achievement and demographic variables and their impact on 
student achievement in the Wissahickon School District* Because the tradi- 
tional reliance on "measures of the average" hides more than it reveals, the 
project is designed to develop ways for administrators and teachers to go 



beyond reporting average score., of student progress. By enabling educators to 
disaggregate individual experienoe from the total population it is possible to 
better understand the impact of district initiatives on individual students or 
sub-groups • 

Research on effective schools, including the research fueling second 
wave restructuring reforms, argues that careful monitoring, analysis, and 
discussion of student progress and program quality are central to improving 
education. Such monitoring, analysis, and discussion requires that: 

• districts collect data on students and their educational 

experiences, 

• data is accessible in a form that can be manipulated, and 

• educators have the necessary tools to manipulate that data 

for fruitful analysis. 

School districts collect a wealth of information about students and 
their academic performance. Much of this data, however, is dispersed through 
out the district. Some data is ir. the central office main frame computer and 
is used for operational decision-making. Some data is in individual student 
files located in schools. Still other data can be found in other offices — 
subject area supervisors, guidance offices, program administrators, etc. 

This dispersion of data is a key factor limiting the usefulness of 
student data. Other limiting factors include the inability to integrate the 
various data kept in different forms, and the lack of analytic tools with which 
to manipulate and interpret data. Moreover, standards are ill-defined. Most 
districts do not have specific indicators of educational and instructional 
expectations for programs, teachers, and students. 

Administrators and teachers alike knew that improving the quality of 
eductioa requires additional information about the impact of education programs 
on student experience. They know they need someth;.ng better than averages, 
rates, and percentages to help guide curricular and program review. Summary 
statistics, a product of an earlier age of office and scoring equipment, will 
no longer suffice. 

The 1980s have 'rought the emergence of computer technologies which 
are flexible, versatile, relatively easy to program, and relatively 
inexpensive. These technologies have brought decision support systems to large 
and small organizations. These technologies are being used by the Wissahickon 
School District in the financial area. We propose to begin to use them in the 
instructional areas. These contemporary and developing technologies - in 
computers, scoring equipment, and in understanding both quantitative and 
qualitative data — provide us with access to better data while enabling us to 
develop better analytic techniques to examine a range of program and curricular 
issues. 

To this end, the Wissahickon School District and the University of 
Pennsylvania Graduate School of Education's Center for School Study Council 
decided to collaborate in a project to develop ways for administrators and 
teachers in the district to more clearly analyze the* impact of district initia- 
tives on student experience. 



Educators* Traditional Use of Data 

Data collected by school districts has primarily been used to address 
problems of particular students or to satisfy federal, state, and local report- 
ing requirements. These reports typically require use of summary statistics: 
numbers of students meeting certain requirements, norms and average test 
scores, attendance rates, etc. Much of this data is available in machine- 
readable form, structured to meet reporting requirements. But computer report- 
ing formats, once set, c^o not permit the flexible manipulation needed for 
ongoing and careful analysis of student experience. 

Moreover, a reliance on summary statistics such as norms hides more 
than it reveals. For example. Figure 1 presents four tables of possible data 
on ten students. For this example, let's imagine that "X" represents scores on 
a math achievement test and "Y" represents scores on a reading achievement 
test. A quick scan of the data suggests the variability of student scores. 

Figure 1 1 
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Yet, as Figure 2 indicates, the summary statistics — means, the 
regression line, the correlation coefficient, etc. for all four tables are 
identical. Attention to such summary statistics would, thtn, mislead one into 
believing that all four groups of students are similar. 

Figure 2^ 

N = 11 

mean of X's = 9.0 
mean of Y's - 7,5 

equation of regression line: Y = 3 + 0.5X 
standard error of estimate of slope = 0.118 
t = 4.24 

sum of squares X - X 110.0 
regression sum of squares = 27,50 
residual sum of squares of Y = 27.50 
correlation coefficient = .82 
r2 « .67 
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Alternatively, Figure 3 draatatically reveals the variety of student 
experience of the four sets of data in Figure 1 . Analysis of those graphs 
suggests the need for different kinds of interventions in each case. 



Figure 3^ 
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The need, therefore, seems obvious for the development of a system 
which permits educators to graphically represent student experience in order to 
effectively reveal individual, sub-group, and total group relationships. This 
project is unique in that it brings graphic displc*' capabilities to the 
analysis of achievement and demographic variables. 

Proposed and Projected Use of Data 

The next step is to develop techniques of representing available data 
in ways that support more careful analysis. These techniques must enable 
administrators and teachers to raise questions about; 

• the general population; 

• specific sub-groups which emerge so educators can understand 

the performance of a student or group of students; and 

• aspects of program offerings for specific students or 

groups of students. 
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The Technique; ^*Display Analysis" 

The techniques we have de/e]oped fall under the rubric ^'display 
analysis." "Display analysis" consists of a number of techniques for 
displaying one or more variables for a target population. These displays 
enable educators to analyze the fit between programs, the pupil population, and 
various sub-groups. Further investigation is then possible using additional 
data — including the personal knowledge staff have of students to better 
understand student progress. (See Figures 4, 4-A, 4-C, 4-D. ) 

Even a very cursory analysis of these five charts provides very 
interesting and important questions. As examples: 

• Why is there a greater variability in the relationship of 
GPA and Achievement Test results for males (Fig. 4-A) than 
females (Fig. 4-B)? 

• Why does the White sub-population show a greater tendency 
to vary in the direction of higher achievement score in 
relation to GPA (Fig. 4-C)? 

• Why does the Black sub-population show a greater tendency 
to vary in the direction of higher GPA in relation to 
achievement scores (Fig. 4-D)? 

• What is the experience of those students who have done so 
well in one measure and average or less in the other (Fig. 4)? 

Display analysis offers the opportunity to visually recognize these 
experiences, to ask questions about them, and to pursue answers based on 
individual experience rather than an average of those experiences. These 
figures are intended to serve as examples and are, therefore, relatively 
simple. Even in ^.his form, however, educators in the pilot project spent 
several hours analyzing them and developing questions for further study. The 
number and kind of such displays are limited only by the data in the file and 
the ability to ask questions. 



Analysis Leading to Interventions 

It is important to realize that "display analysis" does more than 
answer questions. It raises questions and, in the process, supports the 
development of inquiry activities. By enabling educators to list and to graph 
student data on criteria the educators select, it reveals patterns — i.e., 
clusters of students as well as exceptions. Display analysis equips educators 
to see that one or more students who are similar on some educationally relevant 
criter:'.a, are divergent on other important criteria. In these and other ways, 
it supports developing inquiry strategies to discover "critical issues," 
district specific variables or indicators that account for similarities and 
disparities in student experiences. These critical issues are first steps in 
developing educational improvement interventions. 
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Figure 4 

Relationship of Grade Point Average (GPA87) to Percentile Scor^ on Total 
Battery (PR16) of Metropolitan Achievement Test for Selected Student Popula- 
tion. 
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Figure 4-A 
Figure 4 for Male sub-population. 
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Figure 4-B 
Figure 4 for Female sub-population. 
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Figure 4-C 
Figure 4 for White sub-population. 



110- 

ICO- 

( 

I 

I 

I 

30 -I 

■7fj 

50- 
40- 

i 

10 
'2.0 



^ A 



.'4 



A 



V V 



ii'pi-i I ■ A i iijViYfiLi .A 



0.5 1.0 1.5 2.0 2.5 3.0 3.5 



mi 



ERIC 



10 

12 



I 



Fie ire 

Figure 4 for Black sub-population» 
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The Need for Training 

But the use of these tools not intuitive. Educators who have 
become accustomed to discussing means and norms need help refocusing their 
attention to patterns of individual scores. Educators who have become 
accustomed to providing statistical answers need help developing skill in using 
statistics to raise more probing questions. Thus, in addition to developing 
the longitudinal student database, we are continually developing and refining 
tools to analyze data on student educational experiences. We are also 
developing model inquiry strategies that lead to educational interventions. 
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CONCLUSION 

We have developed a flexible and responsive computing environment. 
Success is understood in terms of the actual use of the system in individual 
and organizational decision making. Our next step is to work closely with 
district staff on the kinds of data analysis and reports they requested. in 
the process, new needs will emerge as they modify their requests as they begin 
to explore the data with us. 

Our successes to date have been built upon our ability to respond 
quickly and flexibly to questions and suggestions. Work sessions are inter- 
active, where participants see, act, and react to system structure and output. 
This enables users to follow-up a line of interest and to leave the session 
with information to apply as is to actual work activity. This also teaches 
problem-solving techniques to all participants. The proj ect team plays a 
crucial role in teaching users how to manipulate the technology to keep up with 
the imagination and in raising problem questions. 

Developing these new perspectives requires a great deal of individual 
and organizational learning. We have seen that, once the usefulness of indi- 
vidual student data, aggregated at various levels, is understood, it becomes a 
powerful means of generating probing questions about program structure, con- 
tent, and outcomes. 

We expect that the system will be refined ar new uses are developed. 
Thus, new kinds of data will be added, new ways of presenting that data will be 
developed, and reporting forms will be designed to meet the ongoing and evolv- 
ing needs of users. 

Finally, it appears that the disaggregation of data, combined with 
the flexibility of data manipulation in a graphic presentation offers tremen- 
dous potential to help the practicing educator better understand the experience 
of students. Attempts to explain student experience have encouraged educators 
in this project to develop better inquiry and analysis skills. There is a high 
degree of awareness by project participants that there is less likelihood that 
students will be "lost in the data shuffle" using the display analysis tech- 
nique. Most importantly, educators in the project feel empowered for the first 
time to purposefully utilize the available student data, to analyze it and make 
sense of it. The greatest promise in the project does not lie in the answers 
which we have developed, but rather in the ever improving quality of questions. 
Regardless of the data available, the value of the process is only as great as 
the insights and questions brought by the project participants. In that 
regard, we believe that we have experienced success, and that the potential for 
further development continues to increase. 
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FOOTNOTES 

^Tufte, Edward R. The visual display of quantitative information , 
Cheshire, CT: Graphics Press, 1983. 

2ibid. 

3lbid. 



14 



o 16 

ERIC 



